Let us not get side-tracked in this discussion. Whether or not to include any portion of Twisted into Python 2.6 is well past being a reasonable question; 2.6 alpha 1 has been released. It's a question as to whether someone with commit access can or will commit the patch as posted, run the tests to verify that they aren't broken, and perhaps actually look at the code to verify that we (Giampaolo and I) aren't insane. Mind you, I've been using very similar variants of this patch for months; it fixes outstanding bugs, improves performance, makes the handle* interfaces more consistent, and even offers a 'sample' implementation of a basic protocol in the source (for those who are willing to look). Do we want to fix asyncore/asynchat with work that has already been done and tested?
If you want a reason as to why twisted shouldn't *replace* asyncore/asynchat, I'll give you a few: As stated previously by Guido and others (please see previous threads about this over the course of the last 4 years), asyncore/asynchat provide a more or less minimal interface for asynchronous sockets in Python. Any module/package that seeks to implement asynchronous sockets will need to, at least, implement that much. Asyncore/asynchat at present will play nicely with any event loop available, given certain caveats*. Further, if someone spends a half hour reading the source of a reasonably written asyncore server/client, they should understand the basics and be able to begin using it directly (see any one of the simple echo variants).
As to whether twisted should be added to the standard library at some point in the future as a "this is better than asyncore in every way, use this instead"; that is a different discussion, not related to 2.6 (perhaps not even related to the 2.x series at all, depending on the future of 2.x).
- Josiah
* If your application strictly responds to socket IO, then implement your application as part of handle_* methods. If your application does more, then call asyncore.poll() often enough for data to be handled sufficiently fast. If neither offer sufficient performance or flexibility (maybe something like bittorrent + wxPython), use one thread for your GUI, one thread for your sockets, and use Queue.Queue() or some other threadsafe message delivery method.
On Mon, Mar 24, 2008 at 3:18 PM, A.M. Kuchling <a...@amk.ca> wrote: > On Mon, Mar 24, 2008 at 10:04:20PM +0100, Thomas Wouters wrote: > > I remember only a couple of dissenting voices, and only a small number of > > participants. Of the dissenting voices, I do not recall any actual arguments
> Weren't some of those dissenting voices the Twisted developers, though?
On Mon, Mar 24, 2008 at 03:51:56PM -0700, Josiah Carlson wrote: > reasonable question; 2.6 alpha 1 has been released. It's a question > as to whether someone with commit access can or will commit the patch > as posted, run the tests to verify that they aren't broken, and > perhaps actually look at the code to verify that we (Giampaolo and I) > aren't insane. Mind you, I've been using very similar variants of
I think we should just give you commit access so that you can commit changes to asyncore/asynchat yourself; it doesn't seem as if any of the committers use asyncore enough to check patches for it.
On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...@gmail.com> wrote: > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > summarize them by splitting patches which solve bugs and patches which > > add enhancements or functionalities.
> === Patches for existing issues ===
> - 1736190 which includes fixes for the following issues among other > improvements: > - 1063924 (asyncore should handle ECONNRESET in send()) > - 1736101 (asyncore should handle ECONNABORTED in recv()) > - 760475 (handle_error() should call handle_close() instead of > close()) > - 1740572 (refill_buffer() should call handle_close() rather than > close()) > - 777588 (wrong "connection refused" behavior on Windows) > - 889153 (wrong connect() behavior) > - 953599 (asyncore misses socket closes when poll is used) > - 1025525 (asyncore.file_dispatcher should not take fd as argument)
> - 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ > parameters inconsistency) > - 1541 (Bad OOB data management when using asyncore with > select.poll()) > - 2073 (asynchat push always sends 512 bytes (ignoring > ac_out_buffer_size))
> === Open issues with no patches (need review) ===
> - 658749 (asyncore connect() and winsock errors) > - 1161031 (neverending warnings from asyncore)
> === Enhancements & new features ===
> - 1641 (add delayed calls feature) > - 1563 (conversion to py3k and some other changes)
That's a lot of patches. My immediate concern is that test_asynchat is very flaky and fails often. Sometimes it causes other tests to fail. Is there a patch that addresses this? If you need examples, just look through the buildbot mails that mention test_asynchat in: http://mail.python.org/pipermail/python-checkins/
Some platforms have more problems than others, but almost all platforms have failed test_asynchat at one point or another.
Please break up the patches into 2 sets and prioritize the patches with the set.
Set #1: Patches that have a test and doc updates if required Set #2: Patches that don't have a test or doc updates as required
For the patches that fall into Set #1, list them in priority order. Top priority would be a problem that fixes failures seen in the buildbots. Next priority would go to the patches that solve more serious problems. Post the results here. I will try to look at them.
For every patch you list, make sure that it conforms to the proper style (e.g, PEP 8) and is essentially perfect and ready for inclusion. This means that there is a single file to download that contains all the modifications. The changes are appropriately commented, lines are less than 80 characters, etc. One of the modifications should be an entry in Misc/NEWS.
On Tue, Mar 25, 2008 at 11:26 PM, Neal Norwitz <nnorw...@gmail.com> wrote: > Any reason this was sent just to me and not the list?
Because gmail only replies to the sender by default. I need to remember to cc python-dev when I reply (I used the same email client for 8 1/2 years, remembering the quirks of gmail may take some time).
> On Tue, Mar 25, 2008 at 10:34 PM, Josiah Carlson > <josiah.carl...@gmail.com> wrote:
> > On Tue, Mar 25, 2008 at 9:00 PM, Neal Norwitz <nnorw...@gmail.com> wrote: > > > On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > > > > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...@gmail.com> wrote: > > > > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > > > > summarize them by splitting patches which solve bugs and patches which > > > > > add enhancements or functionalities.
> > > > === Patches for existing issues ===
> > > > - 1736190 which includes fixes for the following issues among other > > > > improvements: > > > > - 1063924 (asyncore should handle ECONNRESET in send()) > > > > - 1736101 (asyncore should handle ECONNABORTED in recv()) > > > > - 760475 (handle_error() should call handle_close() instead of > > > > close()) > > > > - 1740572 (refill_buffer() should call handle_close() rather than > > > > close()) > > > > - 777588 (wrong "connection refused" behavior on Windows) > > > > - 889153 (wrong connect() behavior) > > > > - 953599 (asyncore misses socket closes when poll is used) > > > > - 1025525 (asyncore.file_dispatcher should not take fd as argument)
> > > > - 1641 (add delayed calls feature) > > > > - 1563 (conversion to py3k and some other changes)
> > > That's a lot of patches. My immediate concern is that test_asynchat > > > is very flaky and fails often. Sometimes it causes other tests to > > > fail. Is there a patch that addresses this? If you need examples, > > > just look through the buildbot mails that mention test_asynchat in: > > > http://mail.python.org/pipermail/python-checkins/
> > No, it's one patch. All of the fixes were performed mostly by myself > > last spring, tested and verified in personal servers, then re-used by > > Giampaolo in his async ftp server (which pointed out a few small bugs, > > which have been fixed).
> > > Some platforms have more problems than others, but almost all > > > platforms have failed test_asynchat at one point or another.
> > Certainly that is the case. And according to my reading of a few > > buildbot failures, aside from someone breaking asyncore itself, the > > failures seem to stem from the test being unable to create a port to > > listen on in order to test the server/client functionality. This is a > > buildbot configuration issue (per host), not an asyncore issue.
> That was the case a long time (~3? months) ago, but hasn't been the > case recently. See my recent message about the release.
I'll look for it tomorrow. For reference, searches of 'site:mail.python.org test_asynchat failure buildbot' only seem to produce the socket listen error. If there is a better incantation to get google to produce the proper errors (and/or a link), I would appreciate the help.
> > > Please break up the patches into 2 sets and prioritize the patches > > > with the set.
> > > Set #1: Patches that have a test and doc updates if required > > > Set #2: Patches that don't have a test or doc updates as required
> > > For the patches that fall into Set #1, list them in priority order. > > > Top priority would be a problem that fixes failures seen in the > > > buildbots. Next priority would go to the patches that solve more > > > serious problems. Post the results here. I will try to look at them.
> > > For every patch you list, make sure that it conforms to the proper > > > style (e.g, PEP 8) and is essentially perfect and ready for inclusion. > > > This means that there is a single file to download that contains all > > > the modifications. The changes are appropriately commented, lines are > > > less than 80 characters, etc. One of the modifications should be an > > > entry in Misc/NEWS.
> > I lied earlier; really there are two patches. The first is a patch to > > asyncore.py and asynchat.py . It addresses those bugs that Giampaolo > > has listed, it is tested, and works. The second patch is to update > > the documentation to mention the sample methods in asynchat for use as > > examples, as well as any other changes to the language used in the > > documentation that I had made last spring, but which are out of date > > from my posting of the original patch. I can update the documentation > > in the next week.
> Can you provide a link to the patches? Do the patches include changes > to test_asyncore and test_asynchat? The next release is April 2. I > would like to commit any patches before Monday to ensure they are > stable. Can you get me the patches by this Saturday?
See http://bugs.python.org/issue1736190 for an updated patch for the modules. The current test cases pass without issue, though we may want to add tests, which I need to look at the original patch and the original file from which it was created against, then compare it with the most recent changes to the tests from Facundo last June or July.
I should have the time to get patches for tests and documentation by Monday.
I haven't really had time to update the tests/documentation, but again, I wasn't experiencing any strange test failures with asyncore/asynchat, nor have I been able to find the buildbot failures that you are referring to. Could someone please link the failures that are not related to being unable to discover a port number?
According to the release schedule, we should have at least a couple more months for documentation and tests to be updated (I can get patches ready for alpha 3).
<josiah.carl...@gmail.com> wrote: > On Tue, Mar 25, 2008 at 11:26 PM, Neal Norwitz <nnorw...@gmail.com> wrote: > > Any reason this was sent just to me and not the list?
> Because gmail only replies to the sender by default. I need to > remember to cc python-dev when I reply (I used the same email client > for 8 1/2 years, remembering the quirks of gmail may take some time).
> > On Tue, Mar 25, 2008 at 10:34 PM, Josiah Carlson > > <josiah.carl...@gmail.com> wrote:
> > > On Tue, Mar 25, 2008 at 9:00 PM, Neal Norwitz <nnorw...@gmail.com> wrote: > > > > On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > > > > > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...@gmail.com> wrote: > > > > > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > > > > > summarize them by splitting patches which solve bugs and patches which > > > > > > add enhancements or functionalities.
> > > > > === Patches for existing issues ===
> > > > > - 1736190 which includes fixes for the following issues among other > > > > > improvements: > > > > > - 1063924 (asyncore should handle ECONNRESET in send()) > > > > > - 1736101 (asyncore should handle ECONNABORTED in recv()) > > > > > - 760475 (handle_error() should call handle_close() instead of > > > > > close()) > > > > > - 1740572 (refill_buffer() should call handle_close() rather than > > > > > close()) > > > > > - 777588 (wrong "connection refused" behavior on Windows) > > > > > - 889153 (wrong connect() behavior) > > > > > - 953599 (asyncore misses socket closes when poll is used) > > > > > - 1025525 (asyncore.file_dispatcher should not take fd as argument)
> > > > > - 1641 (add delayed calls feature) > > > > > - 1563 (conversion to py3k and some other changes)
> > > > That's a lot of patches. My immediate concern is that test_asynchat > > > > is very flaky and fails often. Sometimes it causes other tests to > > > > fail. Is there a patch that addresses this? If you need examples, > > > > just look through the buildbot mails that mention test_asynchat in: > > > > http://mail.python.org/pipermail/python-checkins/
> > > No, it's one patch. All of the fixes were performed mostly by myself > > > last spring, tested and verified in personal servers, then re-used by > > > Giampaolo in his async ftp server (which pointed out a few small bugs, > > > which have been fixed).
> > > > Some platforms have more problems than others, but almost all > > > > platforms have failed test_asynchat at one point or another.
> > > Certainly that is the case. And according to my reading of a few > > > buildbot failures, aside from someone breaking asyncore itself, the > > > failures seem to stem from the test being unable to create a port to > > > listen on in order to test the server/client functionality. This is a > > > buildbot configuration issue (per host), not an asyncore issue.
> > That was the case a long time (~3? months) ago, but hasn't been the > > case recently. See my recent message about the release.
> I'll look for it tomorrow. For reference, searches of > 'site:mail.python.org test_asynchat failure buildbot' only seem to > produce the socket listen error. If there is a better incantation to > get google to produce the proper errors (and/or a link), I would > appreciate the help.
> > > > Please break up the patches into 2 sets and prioritize the patches > > > > with the set.
> > > > Set #1: Patches that have a test and doc updates if required > > > > Set #2: Patches that don't have a test or doc updates as required
> > > > For the patches that fall into Set #1, list them in priority order. > > > > Top priority would be a problem that fixes failures seen in the > > > > buildbots. Next priority would go to the patches that solve more > > > > serious problems. Post the results here. I will try to look at them.
> > > > For every patch you list, make sure that it conforms to the proper > > > > style (e.g, PEP 8) and is essentially perfect and ready for inclusion. > > > > This means that there is a single file to download that contains all > > > > the modifications. The changes are appropriately commented, lines are > > > > less than 80 characters, etc. One of the modifications should be an > > > > entry in Misc/NEWS.
> > > I lied earlier; really there are two patches. The first is a patch to > > > asyncore.py and asynchat.py . It addresses those bugs that Giampaolo > > > has listed, it is tested, and works. The second patch is to update > > > the documentation to mention the sample methods in asynchat for use as > > > examples, as well as any other changes to the language used in the > > > documentation that I had made last spring, but which are out of date > > > from my posting of the original patch. I can update the documentation > > > in the next week.
> > Can you provide a link to the patches? Do the patches include changes > > to test_asyncore and test_asynchat? The next release is April 2. I > > would like to commit any patches before Monday to ensure they are > > stable. Can you get me the patches by this Saturday?
> See http://bugs.python.org/issue1736190 for an updated patch for the > modules. The current test cases pass without issue, though we may > want to add tests, which I need to look at the original patch and the > original file from which it was created against, then compare it with > the most recent changes to the tests from Facundo last June or July.
> I should have the time to get patches for tests and documentation by Monday.
> I haven't really had time to update the tests/documentation, but > again, I wasn't experiencing any strange test failures with > asyncore/asynchat, nor have I been able to find the buildbot failures > that you are referring to. Could someone please link the failures > that are not related to being unable to discover a port number?
> According to the release schedule, we should have at least a couple > more months for documentation and tests to be updated (I can get > patches ready for alpha 3).
When you get the patches with tests and doc, I'll be happy to check in.
> > I haven't really had time to update the tests/documentation, but > > again, I wasn't experiencing any strange test failures with > > asyncore/asynchat, nor have I been able to find the buildbot failures > > that you are referring to. Could someone please link the failures > > that are not related to being unable to discover a port number?
> > According to the release schedule, we should have at least a couple > > more months for documentation and tests to be updated (I can get > > patches ready for alpha 3).
> When you get the patches with tests and doc, I'll be happy to check in.
> n
I have updated the documentation, and as much of the tests as was required to pass on my Windows XP machine. In looking at the buildbots, I'm not seeing any more common issues. Unfortunately, I also hit the urls provided above too late, and I wasn't able to see the actual issues (if they still persist).
If possible, the syntax in the documentation that I added should be checked, as I needed to convert from my older TeX docs to the new ReST doc format.
I have attached the patch file, and am in the process of regaining access to the bug tracker. In the mean time, Giampaolo will be posting the patch to the tracker in issue 1736190.
|
